Haptic feedback for interactions with foldable-bendable displays

ABSTRACT

A flexible device includes a bendable-foldable display that has bendable flaps connected by a hinge. The display has sensors for detecting a folding characteristic between the at least two flaps and for detecting a bending characteristic in at least one flap. The display has a haptic system with haptic output devices, where the haptic system receives input from the sensors indicating deformation of the bendable-foldable display device. A flexible device also includes bendable, foldable, or rollable displays that have sensors and actuators to augment user interaction with the device. Based on one or more measurements provided by the input, the haptic system interprets the input to determine deformation characteristics of the bendable-foldable display device. The haptic system generates haptic feedback based on the deformation characteristics.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority of Provisional Patent Application Ser.No. 61/816,605, filed on Apr. 26, 2013, the contents of which is herebyincorporated by reference.

FIELD

One embodiment is directed to a haptically-enabled device. Moreparticularly, one embodiment is directed to a haptically-enabledfoldable-bendable display.

BACKGROUND INFORMATION

Electronic device manufacturers strive to produce a rich interface forusers. Conventional devices use visual and auditory cues to providefeedback to a user. In some user interfaces, kinesthetic feedback (suchas active and resistive force feedback) and/or tactile feedback (such asvibration, texture, and heat) are also provided to the user, moregenerally known collectively as “haptic feedback” or “haptic effects.”Haptic feedback can provide cues that enhance and simplify the userinterface. For example, vibration effects, or vibrotactile hapticeffects, may be useful in providing cues to users of electronic devicesto alert the user to specific events, or provide realistic feedback tocreate greater sensory immersion within a simulated or virtualenvironment.

In order to generate vibration or other effects, many devices utilizesome type of actuator or haptic output device. Known actuators used forthis purpose include an electromagnetic actuator such as an solenoidactuator, an Eccentric Rotating Mass (“ERM”) actuator in which aneccentric mass is moved by a motor, a Linear Resonant Actuator vibrationmotor (“LRA”), electro-active polymer actuator, or a piezoelectricactuator, etc. Kinesthetic actuators may be used to provide hapticfeedback through mechanical movement of a device.

SUMMARY

One embodiment is a bendable-foldable display with at least two flapsconnected by a hinge, wherein each flap is bendable. The display hassensors for detecting a folding characteristic between the at least twoflaps and for detecting a bending characteristic in at least one flap.The display has a haptic system with haptic output devices, where thehaptic system receives input from the sensors indicating deformation ofthe bendable-foldable display device. Based on one or more measurementsprovided by the input, the haptic system interprets the input todetermine deformation characteristics of the bendable-foldable displaydevice. The haptic system generates haptic feedback based on thedeformation characteristics.

One embodiment is a flexible display that is at least bendable,foldable, or rollable. The display has sensors for detecting adeformation or attempted deformation of the display. The displayincludes a haptic system with one or more haptic output devices. Thehaptic system, receives input from the one or more sensors indicatingdeformation of the flexible display device. Based on the input, thehaptic system interprets the input to determine deformationcharacteristics of the flexible display device. The haptic systemgenerates haptic feedback based on the deformation characteristics.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a haptically-enabled system in accordance withone embodiment of the present invention.

FIG. 2 illustrates an action associated with a deformationcharacteristic in one embodiment.

FIG. 3 illustrates an action associated with a deformationcharacteristic in one embodiment.

FIG. 4 illustrates an action associated with a deformationcharacteristic in one embodiment.

FIG. 5 illustrates an action associated with a deformationcharacteristic in one embodiment.

FIG. 6 is a flow diagram illustrating the providing of haptic feedbackin a foldable-bendable device in accordance with some embodiments.

FIG. 7 is a block diagram of a haptically-enabled system in accordancewith one embodiment of the present invention.

FIG. 8 is a flow diagram illustrating the augmenting of haptic feedbackin a flexible device in accordance with some embodiments.

FIG. 9 illustrates an action associated with a deformationcharacteristic in one embodiment.

FIG. 10 is a flow diagram illustrating the providing of haptic feedbackin a flexible device in accordance with some embodiments.

FIG. 11 is a flow diagram illustrating the providing of haptic feedbackin a flexible device in accordance with some embodiments.

FIG. 12 is a flow diagram illustrating the providing of haptic feedbackin a flexible device in accordance with some embodiments.

DETAILED DESCRIPTION

Flexible displays are one of the most anticipated features of the nextgeneration of smart phones, and have attracted a lot of attention inindustry and academia during the past years. Flexible displays canenable a wide array of interactions. Foldable displays can have aseamless hinge or the like without causing a break in pixel presentationin a display. Rollable displays can either be flexible or conform to aparticular curved shape. Rollable displays can also be understood as abendable display in a single direction. Haptic feedback can beincorporated into these devices.

One embodiment is a system that provides haptic feedback associated withgestures and manipulation detected on a simultaneously foldable andbendable display device. A user can interact with an application runningon the device by deforming the display, including by bending or warpingone or more flaps or by changing the angle of the opening between theflaps. The system can characterize the deformation to interpret it intoa user's action or gesture and provide an appropriate haptic feedbackresponse. In one embodiment, for a gesture similar to flipping the pageof a book in an e-reader application, the display can change to the nextpage and haptic feedback can provided to signify the page was changed onthe display. In one embodiment, for a gesture similar to fanning abook's edge, the display can change to a page that is multiple pagesaway from the current page and haptic feedback can be provided tosignify that multiple pages were passed prior to displaying the changedpage. In one embodiment, kinesthetic feedback may make the bendablesurface more rigid or the bendable surface may be naturally fairlyrigid. Thus, a user can apply pressure without actual deformationoccurring. The sensory system can be capture the force/pressure appliedby the user and still provide a vibrotactile or other type of “hapticsresponse” to such interactions by the user, i.e. force applied but notto the point of deformation.

In another embodiment, the system can present multiple objects side byside, on each flap of the bendable-foldable display. The system can usephysical property information for the presented objects to providekinesthetic feedback in response to a user interacting with the flaps tochange the bendability (stiffness) characteristics of the displayaccording to the physical property information of the objects.

In another embodiment, for a gesture similar to the closing of a book,the system can detect the action of closing the flaps together and priorto the complete closing of the flaps, cause the flaps to close andprovide a haptic feedback of a snapping sensation, as of a book closing.In one embodiment, for a gesture similar to the opening of a book, thesystem can detect the action of opening the flaps apart and prior to thecomplete opening of the flaps, cause the flaps to open and provide ahaptic feedback of a snapping sensation, as of a book opening.

Another embodiment is a system that provides augmented haptic feedbackassociated with gestures and manipulation detected on a flexibledisplay. Enhancement of the haptic experience can be accomplished byaugmenting the natural haptics related to continuous (analogue)deformation input dimensions with flexible displays through programmablevibrotactile and kinesthetic haptics.

Another embodiment is a system that provides kinesthetic haptic feedbackassociated with a flexible display to adjust the elasticity ordeformability of the display based on the target interaction. Someinteractions and functionalities may require the screen to be stiffer,or even rigid, while others are better facilitated by higherflexibility/softness of the interface. Embodiments describe aprogrammable haptic stiffness control mechanism to adjust the elasticityor deformability of the display depending on the target interaction.

Another embodiment is a system that provides shape changing functions ofa flexible display. The purposes for taking a particular shape caninclude the ability to take the form that is more appropriate for acertain functionality or application and thus facilitates the userinteractions with the interface; to display certain information throughits geometric shape; to enable functionalities that are only feasiblewith a certain physical shape; or in case the device is serving as aphysical representation of a virtual object, to deliver a more realisticsimulation of the target object by mimicking its shape.

FIG. 1 is a block diagram of a haptically-enabled system 10 inaccordance with one embodiment of the present invention. System 10includes a touch sensitive foldable-bendable surface 11 or other type ofuser interface mounted within a housing 15, and may include mechanicalkeys/buttons 13 and a hinge 14 between the two flaps. Internal to system10 is a haptic feedback system that generates vibrations on system 10.In one embodiment, the vibrations are generated on touch surface 11.

The haptic feedback system includes a processor or controller 12.Coupled to processor 12 are a memory 20 and an actuator drive circuit16, which is coupled to an actuator 18. Actuator 18 can be any type ofDirect Current (“DC”) motor, including without limitation an EccentricRotating Mass (“ERM”), a Linear Resonant Actuator vibration motor(“LRA”), a piezoelectric motor, or a solenoid actuator. In addition toor in place of actuator 18, system 10 may include other types of hapticoutput devices (not shown) that may be non-mechanical or vibrotactilehaptics devices such as devices that generate electrostatic friction(“ESF”), ultrasonic surface friction (“USF”), devices that induceacoustic radiation pressure with an ultrasonic haptic transducer,devices that use a haptic substrate and a flexible or deformable surfaceor shape changing devices and that may be attached to a user's body,devices that provide projected haptic output such as a puff of air usingan air jet, devices that provide electrical muscle stimulation, etc.

Processor 12 may be any type of general purpose processor, or could be aprocessor specifically designed to provide haptic effects, such as anapplication-specific integrated circuit (“ASIC”). Processor 12 may bethe same processor that operates the entire system 10, or may be aseparate processor. Processor 12 can decide what haptic effects are tobe played and the order in which the effects are played based on highlevel parameters. In general, the high level parameters that define aparticular haptic effect include magnitude, frequency, and duration. Lowlevel parameters such as streaming motor commands could also be used todetermine a particular haptic effect. A haptic effect may be considered“dynamic” if it includes some variation of these parameters when thehaptic effect is generated or a variation of these parameters based on auser's interaction.

Processor 12 outputs the control signals to actuator drive circuit 16,which includes electronic components and circuitry used to supplyactuator 18 with the required electrical current and voltage (i.e.,“motor signals”) to cause the desired haptic effects. In instances wherethe haptic effects correspond to the playback of a multimedia file, suchas a video file, processor 12 may provide the haptic control signal tothe haptic drive circuit. System 10 may include more than one actuator18, and each actuator may include a separate drive circuit 16, allcoupled to a common processor 12. Memory device 20 can be any type ofstorage device or computer-readable medium, such as random access memory(“RAM”) or read-only memory (“ROM”). Memory 20 stores instructionsexecuted by processor 12. Among the instructions, memory 20 includes ahaptic effects module 22 which are instructions that, when executed byprocessor 12, generate drive signals for actuator 18 that provide hapticeffects, as disclosed in more detail below. Memory 20 may also belocated internal to processor 12, or any combination of internal andexternal memory.

Touch surface 11 recognizes touches, such as those provided by a user,and may also recognize any of the position, pressure magnitude, andduration of touches on the surface. The data corresponding to thetouches is sent to processor 12, or another processor within system 10,and processor 12 interprets the touches and in response generates hapticeffect signals. Touch surface 11 may sense touches using any sensingtechnology, including capacitive sensing, resistive sensing, surfaceacoustic wave sensing, pressure sensing, optical sensing, etc. Touchsurface 11 may sense multi-touch contacts and may be capable ofdistinguishing multiple touches and the location of the touches thatoccur at the same time. Touch surface 11 may be a touchscreen thatgenerates and displays images for the user to interact with, such askeys, dials, etc., or may be a touchpad with minimal or no images.

System 10 may include a variety of sensors, such as sensor 17, forsensing interactions with the foldable-bendable display shown in FIG. 1including, among others: strain gauge sensors to measure the deformationmagnitude during interactions, force sensing resistor (“FSR”) sensors tomeasure the force/stress applied to the flexible display structures,multi-touch touch sensors to detect the location of single or multipletouch inputs in a touch-enabled display, multi-touch pressure sensors tomeasure the pressure applied under each touch location,temperature/humidity/atmospheric pressure sensors to captureenvironmental conditions, an accelerometer/gyroscope/magnetometer tocharacterize the motion, velocity, acceleration and orientation of thedisplay, a microphone to capture a user's voice command or environmentalaudio information, and wireless transmitters to receive/transmitinformation from/to other devices wirelessly. The data corresponding tosensor 17 is sent to processor 12, or another processor within system10, and processor 12 interprets the sensor data and in responsegenerates haptic effect signals.

In addition to the actuators discussed above, system 10 may include avariety of actuators for providing vibrotactile or kinesthetic feedbackincluding flexible, semi-rigid, or rigid actuators, including ElectroActive Polymer (“EAP”) actuators, smart fluids actuators, rheologicalfluidic actuators, Macro-Fiber Composite (“MFC”) actuators, Shape MemoryAlloy (“SMA”) actuators, piezo actuators, and Micro-Electro-MechanicalSystem (“MEMS”) actuators.

System 10 may be a handheld device, such a cellular telephone, personaldigital assistant (“PDA”), smartphone, computer tablet, gaming console,vehicle based interface, etc., or may be any other type offoldable-bendable device that includes a haptic effect system thatincludes one or more actuators. The user interface may be a touchsensitive surface, or can be any other type of user interface such as amouse, touchpad, mini-joystick, scroll wheel, trackball, game pads orgame controllers, etc. In embodiments with more than one actuator, eachactuator that has rotational capability may have a different rotationalcapability in order to create a wide range of haptic effects on thedevice, for example each actuator can be controlled individually; alsosome rotational actuators have their axis of rotation at an angle to theaxis of rotation of other rotational actuators. Likewise, in embodimentswith multiple actuators with other capabilities, each actuator can becontrolled individually to exhibit a wide range of haptic effects on thedevice.

In addition to providing user interfacing haptic effects, system 10 mayprovide statically generated haptic effects for playback in system 10along with, for example, a video or audio file.

Flexible displays in general allow for a wide array of novelinteractions by taking advantage of deformation as an additionalinformation entry mechanism. However, the uniform/continuous feedbackresulting from deformation interactions with flexible displays may notprovide intuitive metaphors that relate events occurring digitally tophysical interactions from a non-digital space. In other words, users donot typically interact in a non-digital space with small flat objects ofthe size of phones and tablets, as opposed to actual books andmagazines. Embodiments offer a system for compelling user interactionsand additional information about the digital environment to the userthrough haptics communication channels. Embodiments expand the userinterface design space, enhance the interaction experience with flexibledisplays, and render deformation-based gestures more intuitive andnatural. Moreover, embodiments provide an effective way to translate anddeliver information about the digital world in the form of haptics.

Rather than a single surface flexible display, simultaneously foldableand bendable flexible displays consist of two or more “flaps” (each ofthe subcomponents of flexible displays) which can revolve around singleor multiple hinges relative to the adjacent flaps, while each flap canalso bend/flex (deform locally). The characteristic of being foldableand bendable at the same time enables metaphors that simulate specificuser interaction scenarios in real life. Interactions with booksrepresent one of the most relevant examples of this category. Forexample, while the page “flip” gesture is the result of the“bendability” (local flexibility) of the paper, each single page orcollection of pages will go through a folding motion around the mainspine (hinge) of the book. Embodiments provide methods of using hapticsfeedback to enhance and enrich user interface interactions withsimultaneously bendable and foldable class of displays. Thus,exploitation of programmable vibrotactile and kinesthetic haptics toaugment interactions with such displays can be achieved by introducinghaptics control and haptic augmentation of both folding and localbending/flexing degrees of freedom of the display. Introducing tactilefeedback to increase realism of interaction as well as adding anadditional tactile information layer provides users of these devices theability to interact with their devices more organically.

Particular embodiments are now discussed which can exploit programmablevibrotactile and kinesthetic haptics to augment interactions withbendable foldable displays and draw on metaphors from non-digitalexperiences of users.

The action of turning the page of a real book consists of graduallybending (bendability) the page's corners or edge, while revolving(foldability) it around the spine at the same time. In a book thetactile sensation of the interaction depends on the page material/typeand thickness, as well as the type of binding. In one embodiment, asdepicted in FIG. 2 a user is reading an e-book on a simultaneouslyfoldable and bendable flexible display, consisting of two flexibleflaps. In order to “flip” the page of the e-book, the user slightlybends the lower right corner of the right flap of the display.

FIG. 2 illustrates an action associated with a deformationcharacteristic in one embodiment. A deformation characteristic is aninterpretation of the deformation of the foldable-bendable display. Adeformation characteristic can include the position of the hinge betweenthe bendable flaps and can also include the bending state of the flaps.A deformation characteristic can include raw data associated withsensors measuring the properties of the hinge position or bending state.A deformation characteristic can be a determined state of thebendable-foldable display by interpreting sensor data into a form thatmay have more generalized meaning, such as an interpretation that thefoldable-bendable display is “slightly open, right flap bent strongly.”A deformation characteristic can also be an interpretation of an actionor gesture, such as can be determined by comparing sequential sensorreadings to interpret the sensor readings as “slightly open, openingslowly, right flap strongly bent, right flap bend increasing.”

Continuing with FIG. 2, the left side of FIG. 2, elements 205, 210, 215,and 220 illustrate the side view of a foldable-bendable device with twoflexible flaps. For a user reading a book on the device, to “flip” thepage, at 205, a user's finger comes in contact with the edge of thedevice at either a corner or in the middle of the edge. At 210, theuser's finger begins to apply pressure at the edge, causing adeformation in the bendable surface. At 215, the user's finger appliessideways pressure in the direction of the hinge of the foldable-bendabledisplay, slightly closing the device and further increasing the bendradius. The user's finger can release the edge of the device. The user'sinteraction from 205 through 215 results in different states ofdeformation of the foldable-bendable device. A particular sequence ofdeformations, such as those provided in 205, 210, and 215, can identifya particular gesture or interaction the user is having with thefoldable-bendable device. They system can determine from the sequence ofdeformations which haptic feedback to provide. For example, the systemcan characterize the deformation sequence by interpreting the parametersreceived by sensors, such as sensor 17, to represent a page flip gesturefrom a user. The system can provide haptic feedback for the page flipgesture, providing a tactile experience for flipping a page on ane-reader application, and at 220, after the page flip bring the flapback to its original position from before the page flip. Particulardetails for providing the haptic feedback are discussed in detail below.

Elements 225, 230, and 235 illustrate a user's action from a top viewperspective. Element 225 shows the foldable-bendable device with ahinge, left flap, and right flap. At 230, the device is shown with anarrow or small opening angle between the two flaps. At 235, a user'sgesture puts pressure on the edge of the device to cause a bending ofone flap and a rotation on the opening angle, increasing the openingbetween the flaps. These two movements taken together is a deformationsequence that can be characterized to determine appropriate hapticfeedback.

During this page-flip interaction, a kinesthetic haptic actuator cancontrol the motion of the flap in a way that it not only bends but alsoslightly revolves around the display's hinge, simulating a realisticinteraction similar to a real book page flip. Programmable kinesthetichaptic, through spatiotemporal control of stiffness, can ensure that theuser receives appropriate and realistic tactile feedback during both thebending and folding motion of the flap. In some embodiments, theperceived resistance/stiffness of the bending/folding flap can bemodulated to represent physical properties of the book/paper, such asthe type of the binding, paper type/thickness, type of the media (book,newspaper, thin magazine, etc.) or other metaphorical information, suchas reaching the end of a chapter or end of the book. Further, after thepage content is updated (page flip has taken place), the hapticcontroller can bring back the right flap to its original position, asdepicted in 220. Bringing the right flap back to its original positioncan be triggered either by the user through a certain gestural input(e.g., user starts to lift his finger off the flap), or be doneautomatically as soon as the page content has been updated to the nextpage on the graphical display. Thus, the combination of tactile feedbackon bending and folding can increase the intuitiveness and realism of theinteraction and increase the information communication bandwidth.

In another embodiment of the invention, the user can exploit abending/folding gesture to quickly browse or “leaf” through the pages ofan e-book. The action of leafing through a real book involves fanningthe pages of the book through a simultaneous bending and foldingtechnique with the book, while keeping pressure on the edge of the book,but varying the arch of bending and the angle between the left and rightpages of the book.

FIG. 3 illustrates an action associated with a deformationcharacteristic in one embodiment. At 305, the top view perspective of afoldable-bendable display is depicted with a hinge, a left flap, and aright flap. At 310, in order to leaf/browse through the pages of thee-book, the user may start with the foldable display flaps almost closed(flaps only slightly apart), and then attempt to slightly bend the rightflap (similar to the interaction with a real book). At 315, as the useris folding/bending the display open, similar to what was explained inconnection with FIG. 2, the haptic controller ensures a realistic hapticsensation similar to bending a real book. At 320, the user slightlybends and then releases the flap. Upon being released, the flap movesaway (with a controlled bending and folding motion) from user's fingeruntil the contact between finger and flap is lost. At 325, a hapticscontroller brings the flap back to the position right before it wasreleased, in a way that it touches user's finger again. The flaprepeatedly (and automatically) does this back and forth motion. In eachrepetition the flap slightly touches user's finger, while each instanceof finger-page contact represents a single (or possibly multitude ofpages) browsed through. The haptic creates a sensation that would feelsimilar to pages of a book slipping from under the finger. Thus,browsing or leafing through pages is another relevant metaphor that canbe effectively emulated and haptically augmented to enrich theinteraction based on the described haptic feedback scheme.

In some embodiments, at 320, rather than releasing the flap, the usercan maintain contact with the flap. The rate that pages change from onepage to the next can be controlled by increasing or decreasing the bendof the flap. As the bend radius decreases (increasing the bend), therate of the page flip increases. The opening angle between the two flapscan also be used to define how quickly the pages change on the screen.At 325, instead of the flap coming back to make contact, a detent alongthe edge can house an actuator to brush the fingertip, providing afeeling of pages slipping past the fingertip.

In some embodiments, at 325 when the haptic controller brings the flapback to the position right before it was released, it does not touch theuser's finger again. Pages can cease flipping when contact is not made.Pages can flip until the user interacts with the device again. Pages canflip for a number or percentage of additional pages before stoppingflipping. The flap can repeatedly (and automatically) do the back andforth motion while flipping or can remain stationary.

FIG. 4 illustrates an action associated with a deformationcharacteristic in one embodiment. In some embodiments, system 10 is afoldable-bendable display device used for the purpose of comparingphysical/haptic characteristics of two objects 410 and 420, eachdisplayed on one of the two flaps of the flexible display. System 10stores physical property information about each of object 410 and 420.Using programmable kinesthetic haptics, the bending stiffness/resistanceof each flap can be mapped to the physical properties of thecorresponding object displayed. Vibrotactile haptics can also be used tomodel the physical properties of the corresponding displayed objects asthey are manipulated. Therefore, the user can examine the physicalproperties, and in particular a stiffness of the objects displayed sideby side. As the user manipulates each flap of system 10, the bendingstiffness/resistance of each flap can vary according to the bend pointof the display, the object location on the display, and the physicalproperties information for the object. For example, when an object isstiff at a particular place on the object, then bending the flap at thatpoint will meet with greater resistance than bending the flap at a pointwhere the object is not as stiff. Further, the physical properties ofthe objects can change based on the input. For example, if object 410 isglass, then kinesthetic haptics can make the bendability of the flapstiff where the object is displayed. If the user bends the flap anyway,the image could react by breaking as if it were real glass and providevibrotactile haptics to accentuate the glass breaking effect. Thebending characteristic can be changed to suit the new physicalcharacteristics of broken glass object 410.

FIG. 5 illustrates an action associated with a deformationcharacteristic in one embodiment. In some embodiments, the gesture ofcompletely folding open or folding close of the display can behaptically augmented using tactile feedback, to emulate a snap-close ora snap-open metaphor. This can simulate the haptic feedback experiencewhen opening or closing a real book, newspaper, magazine, etc., such assnapping a book open or closed. In 510, the user folds the displayclosed in order to close an application or close a book. The angle in510, θ, can change toward zero as the flaps are closed. In 515, at someangle, θ′, the deformation characteristic can be interpreted as a bookclosing and kinesthetic actuators can control the angle of the flap totake over the closing of the flaps. At the point where the angle θequals zero or the system determines the flaps are closed, additionalvibrotactile and auditory haptic feedback can be provided that resemblesthe real world scenario of closing a book. In 520, a user folds thedisplay open in order to open an application or open a book. The anglein 520, θ, can change toward 180 degrees as the flaps are opening. In525, at some angle θ′, the deformation characteristic can be interpretedas a book opening and kinesthetic actuators can control the angle of theflap to take over the opening of the flaps. At the point where the angleθ increases from zero to a positive number or the system determines theflaps are opening, additional vibrotactile and auditory feedback can beprovided that resembles the real world experience of opening a book.

FIG. 6 is a flow diagram illustrating the providing of haptic feedbackin a foldable-bendable device in accordance with some embodiments. Insome embodiments, the functionality of the flow diagram of FIG. 6 isimplemented by software stored in memory or other computer readable ortangible medium, and executed by a processor. In other embodiments, thefunctionality may be performed by hardware (e.g., through the use of anapplication specific integrated circuit (“ASIC”), a programmable gatearray (“PGA”), a field programmable gate array (“FPGA”), etc.), or anycombination of hardware and software.

At 605, an application is initiated on the device. The application canbe any software program that operates on the foldable-bendable device.For example, the application can be an e-reader application for readingmaterials that may be traditionally printed on paper, such as books,magazines, paper pads, etc. The application can be configured tospecifically work in an environment that provides haptic feedback on afoldable-bendable display device. At 610, one or more sensors located onthe foldable-bendable display registers a change in properties,indicating a deformation of the display. The deformation can includebending or warping of the flaps or one of the flaps and/or changing theangle between flaps (opening angle). The one or more sensors can eachprovide individual measurements, which can then in turn be analyzed tocharacterize a user's actions at any given time or over a period oftime. At 615, based on the values of the one or more sensors, thedeformation is characterized to establish what type of action a user isperforming on the foldable-bendable device. Values of the one or moresensors can be compared to values of the one or more sensors atdifferent times. For example, the value of a sensor detecting theopening angle between two flaps can be compared to the value of the samesensor one second, two seconds, or n seconds before to help characterizethe deformation. The system can continue to monitor the one or moresensors until the deformation can be characterized. At 620, when thedeformation has been characterized, the system provides haptic feedbackbased on the characterization of the deformation. Such haptic feedbackcan include vibrotactile or kinesthetic feedback. The haptic feedbackcan also be coordinated with visual or audio feedback. For example, in apage flip gesture, a graphic or sound of a flipping page can coincidewith vibrotactile or kinesthetic feedback. The system can continue tomonitor the one or more sensors to model additional deformationcharacteristics.

Bend sensors, such as sensor 17, can detect bending actions in one edgeand its two corners such as bending the edge up, bending a corner down,bending a corner up, bending both corners up and in, bending the edgeback, and bending both corners back and in thereby bending the middleedge upward. One skilled in the art will recognize that other bendingactions can be sensed in a similar manner as these bending actions,including similar bends to other edges and corners and bends to thesurface of the display away from edges and corners.

Bend sensors, such as sensor 17, can measure a voltage response in thesensor over a time period. The voltage response can indicate in thesensor whether no bending is sensed, slight bending is sensed, or strongbending is sensed. These values over time can indicate particulargestures as deformation changes over time. Also, additional sensors,such as sensor 17, can be used to provide additional data referencepoints to obtain bending and deformation characteristics over differentpoints on bendable-foldable display 10.

Fold sensors, such as sensor 17, can detect folding actions at a hinge,in accordance with some embodiments. Types of folds that can be detectedinclude no fold (flat), a symmetrical downward fold, a symmetricalupward fold, a fold from the back flap to the front flap to close, afold from front flap to open the back flap, a fold from the back flap tothe front flap backwards, a fold from the front flap to the back flap toopen in reverse, a continuous fold in an up and down manner, and acomplete fold closed. Forms of folds include a center fold where eachflap is about the same size, a partial fold where each flap may bedifferent sizes so that the hinge appears to be offset from center, andvarious combinations for embodiments with more than two flaps. Touchsensors, such as touch sensor 11, can be used in conjunction with a foldsensor to sense one handed folding, two handed folding, single finger,and multi-finger folding. The data from sensor 17 capturing foldingsensing can be charted over time to determine deformationcharacteristics. Also, additional sensors, such as sensor 17, can beused to provide additional data reference points for bendingcharacteristics to define deformation actions on bendable-foldabledisplay 10.

Different interactions with a bendable-foldable display can map todeformation characteristics similar to a non-digital analogue. Forexample, in a book reading application, the action of folding open aflap of bendable-foldable display 10 may correspond to opening orclosing the cover of a book. The action of sliding a finger down theedge while holding bendable-foldable display 10 slightly open maycorrespond to adjusting the starting point of page retrieval. The actionof bending and releasing the corner of bendable-foldable display 10 maycorrespond to turning a page. The action of gently bending the edge backof bendable-foldable display 10 while folding the flaps slightly moreclosed may correspond to leafing through pages slowly. The action ofgreatly bending the edge back of bendable-foldable display 10 whilefolding the flaps slightly more closed may correspond to leafing throughpages quickly. The action of holding both flaps of bendable-foldabledisplay 10 slightly closed while bending both edges slightly back maycorrespond to placing a book face down. The action of folding the flapsof bendable-foldable display 10 and swiping near the hinge maycorrespond to inserting a bookmark. The action of holding one finger onone flap and sliding another finger down the edge while holding theflaps slightly closed may correspond to turning pages while keeping afinger between pages at a point of interest (finger bookmark). One ofskill in the art will understand that these actions can map to differentfunctions in other applications. For example, an application for a photoalbum may use the same actions to determine different interactions withthe application.

FIG. 7 is a block diagram of a haptically-enabled system 10 inaccordance with one embodiment of the present invention. System 10includes a touch sensitive flexible surface 711 or other type of userinterface mounted within a housing 15, and may include mechanicalkeys/buttons 13. Internal to system 10 is a haptic feedback system thatgenerates vibrations on system 10. In one embodiment, the vibrations aregenerated on touch surface 711. Note that the description above withrespect to FIG. 1 applies equally to FIG. 7, except that the surface 711is flexible so that it can be bendable, foldable, rollable, or anycombination thereof. Perceived physical properties of flexible surface711 can be changed to be formed to be very flexible (like a sheet ofpaper) to nearly rigid (with only a few degrees of flexibility).Embodiments include where the display is partially flexible (rigid insome parts and deformable in some other parts). Embodiments includewhere the display is non-uniformly elastic and can use kinesthetichaptics to provide perceived uniform and consistent elasticity anddeformability. Embodiments also include the reverse, where kinesthetichaptics can provide varying perceived elasticity across a surface withnaturally homogeneous elasticity.

Flexible displays are one of the fundamental building blocks of organicuser interfaces and can potentially enable digital user interactionsthat are more intuitive, realistic, and comparable to those occurring inreal world. Except for the permanently conformed class of bendabledisplays, interactions with flexible displays such as flexible display711 involve deforming (bending, folding and rolling) the display as aninput gesture. Such deformations provide a continuous (analogue)directional information entry mechanism which can be expanded intoseveral input dimensions. For example, a “bending” gesture can beexpanded into bending along the width/length/diagonal, bending thecorners, and so forth.

Deformation-based interactions with flexible displays already offer somelevel of (natural) haptics. For example, in a bendable device withoutany haptic augmentation, the device will provide a natural resistance tobeing bent. However, the natural, uniform, and continuous tactilefeedback resulting from deformation interactions does not carryinformation about the events occurring in the digital world and is notprogrammable. Enhancement of the haptic experience can be accomplishedby augmenting the natural haptics related to continuous (analogue)deformation input dimensions with flexible displays through programmablevibrotactile and kinesthetic haptics.

While natural deformation interactions are similar to those in realworld, synthetic haptics holds the potential to offer an intuitive wayto convey detailed information about the digital environment. Augmentingthe natural/intuitive deformation interactions with syntheticprogrammable tactile feedback allows feedback that translates theinformation about the results/consequences of the deformation input inthe digital world into haptics.

FIG. 8 is a flow diagram illustrating the augmenting of haptic feedbackin a flexible device in accordance with some embodiments. At 805, theflexible device functionality is started. This can include launching anapplication or interface that can haptically respond to the gestures ofa user on a flexible display, such as flexible display 711. At 810, adetection of a gesture occurs on the flexible display by a sensor suchas sensor 17. Gestures can include deforming of the flexible display bybending, folding, or rolling the display. Gestures can include deformingthe flexible display in free space, by for example, waving the displaylike a flag. Gestures can also include touch gestures that are done inconjunction with deforming gestures. Gestures can include pressureapplied to a deformable surface, but without actual deformation.

At 815, vibrotactile or kinesthetic haptics are produced based on thegesture and based on the state of the application or interface on theflexible display. The state of the application or interface provide thecontext to the user about the events occurring in the digital world,i.e., on the display. Vibrotactile haptics can include any effects thatcan be played by one or more vibrotactile actuators in the displaydevice or worn by a user of the flexible display device. Kinesthetichaptics include any effects that produce movement of the flexibledisplay device or alter the perceived physical characteristics of theflexible display.

At 820, the effects of the gesture are carried out on the application orinterface. This includes displaying the haptic effects, e.g., displayinga vibrotactile response; augmenting the gesture, e.g., providingkinesthetic haptic feedback to the user; and changing the hapticperception of the display, e.g., including a vibrotactile cue inconjunction with updating a visual or auditory output of the displaybased on the interaction. For example, an application may be displayinga picture on flexible display 711. Deformation of the display in aparticular way can cause a vibrotactile sensation to provide feedbackindicating that a command was received, a kinesthetic response to makethe display flat again, an auditory feedback to give notice that thepicture displayed is changing, and a visual feedback to advance thepicture displayed to the next picture in a gallery. The system continuesto monitor sensor 17 and other input mechanisms for additional input.Thus, vibrotactile and kinesthetic haptics can be employed to confirmthe completion/accomplishment of an interaction.

In one embodiment, the user can interact with a flexible portabledigital device running an e-book application. The device is equippedwith a flexible (bendable) display interface. In a known application,the gesture of bending the corner of the display can be mapped toleafing through the pages the e-book in the digital environment. Thebendable display can also be equipped with whole-body vibrotactilehaptics functionality. As the user gradually bends the corner of thedisplay, the pages of the e-book flip. The rate of the page flip cancorrelate with the extent of bending/bending angle.

However, neither the magnitude nor angle of deformation nor the bendingstiffness or resistance force (i.e., the natural haptics) provides theuser with a direct one-to-one mapping with the number of pages flippedand/or the rate of the page flip. In some embodiments, the cornerbending gesture can be augmented with vibrotactile programmable detents,where each detent (haptic click) indicates a single (or a certain numberof) page flip(s). The number of tactile detents, therefore, hapticallyinforms the user about the number of pages flipped, and the rate ofdetents presents a tactile indication of the relative pace of browsingthrough pages. Adding this type of vibrotactile feedback to theinterface allows the user to establish a mental model between the hapticsensation (deformation augmented with synthetic haptics) and the event(page flip) occurring in the digital environment.

FIG. 9 illustrates an action associated with a deformationcharacteristic in one embodiment. In 901, a user bends the corner of aflexible (bendable) device. A haptic response is played in the form of avibrotactile feedback. The corresponding interaction mapping, however,depends on the range/threshold of bending magnitude. Four differentmapping schemes are distinguished in FIG. 9. Each stage corresponds to acertain bending angle range, e.g., stage 1 (at 910) if bending anglelies between 0 and 10 degrees, stage 2 (at 920) between 10 to 20degrees, stage 3 (at 930) between 20 to 30 degrees, and stage 3 (at 930)between 30 to 40 degrees, and so forth. Each stage can have a differentfunctionality or mapping associated with it. For example, in a photobrowsing application, bending the corner of the device (back and forth)between 0 to 10 degrees (910) can be mapped to scrolling through a setof pictures. Bending the corner of the device beyond 10 degrees up to 20(920), can allow the user to continuously zoom in/out the pictures. Instage 3 (930), the user can use the bending gesture to adjust thebrightness of the picture. Finally in stage 4 (940), bending input canbe used to alter the transparency of the picture. In this embodiment,the type, strength and pattern of the tactile effects delivered to theuser can be different for each stage, as indicated in the graphsaccompanying each of stages 910, 920, 930, and 940. Haptics not onlyaugments the continuous interaction in each stage, it also helps theuser distinguish between different stages/mapping schemes related tovarious functionalities. Moreover, the transition between the stages canbe haptically marked to provide the user with further tangible insightas to differentiating between the stages. In other words, the user canuse haptic marks to explore among various functionalities withoutnecessarily resorting to audiovisual cues. One skilled in the art willunderstand that although bending angle is used as the range distinctionparameter in the above example, the particular bending angles can bechosen differently and other parameters such as force, etc., could alsobe used for this purpose.

In other embodiments, the continuous “folding” gestures applied to afoldable display can likewise be haptically augmented using programmabledetents as described above. The use of vibrotactile feedback effects isnot limited to detents, and in fact any type of appropriate vibrotactileeffects could be employed to augment the interactions.

Similar to vibrotactile feedback, kinesthetic haptics can offer value insimilar or different contexts, by augmenting the interactions. Examplesof methods to kinesthetic haptics include programmable stiffness display(stiffness control), structural damping (damping factor resistingbending, folding, and other deformation degrees of freedom), etc.Feedback can be a combination of both vibrotactile haptics andkinesthetic haptics.

In some embodiments, the natural or structural stiffness and damping ofthe materials used in flexible displays can be augmented withprogrammable haptic stiffness control to enrich the deformation gesturelanguage and enhance the effectiveness of the interface. For example, inone embodiment, the user can use bending deformation gestures to browsethrough his/her list of contacts. As the user becomes closer the end ofthe list, the haptic stiffness control function can vary the perceivedresistance against the deformation, providing the user with a sense ofwhere he/she is in the list. Once the user reaches the end of the list,the haptic stiffness scheme can create a haptic “barrier” by making thedisplay non-deformable/rigid in one direction, preventing the user frombending further. Likewise, when the user is far from the end of thelist, the display can be made to be less rigid by countering the naturaldeformation resistance of the device using kinesthetic haptics. In otherwords, the range of deformation of the bendable display can becontrolled using haptics.

Further, kinesthetic stiffness/damping control or vibrotactile effectscan be used to haptically mark each item in the list with programmabledetents. For example, a different haptic effect/mark can be used whenthe user passes between different alphabetic groups. The same hapticaugmentation could be applied to folding gesture in a foldable displayinterface.

In one embodiment, the perceived haptic “stiffness” or deformationresistance of the display can be customized and tailored to therequirements of the current application or required functionality. Forexample, a user may intend to use the bending or folding gesture to fastforward a sound track, but in small increments or time steps. Thestiffness of the display can be actively controlled to assist the userto deform (bend or fold) the display in a controlled gradual manner,with the level of bending or folding corresponding to the size of theincrements or time steps either as a fixed size increase or as apercentage of the file length.

Turning back to FIG. 7, another embodiment is a system that provideskinesthetic haptic feedback associated with a flexible display to adjustthe perceived elasticity or deformability of the display based on thetarget interaction. Here a devices natural “elasticity” means the easeof flexion, folding, or rolling that a flexible display has in itsnatural state based on the material properties and build of the device.Although elasticity of the flexible displays sets the stage for enrichedgesture language and advanced intuitive interactions, too muchflexibility may have negative impact on some user interfaceinteractions, and in some cases will render them infeasible. Someinteractions and functionalities may require the screen to be stiffer oreven rigid, such as a screen press, while others are better facilitatedby higher flexibility or softness of the interface. Embodiments describea programmable haptic stiffness control mechanism to adjust theelasticity or deformability of the display depending on the targetinteraction. In other words, depending on the interactions and thetarget function, the bendable interface becomes more stiff if a morerigid interface is required, or become softer (easier to deform) if useris applying a deformation-based gesture.

One of skill in the art will understand, however, that deformation ofthe display may not necessarily occur, and the sensory system cancapture the force/pressure applied by the user. For example, thekinesthetic haptics can render the display quite stiff, in which case asmall amount of force applied by the user will not lead to any physicaldeformation. However, the system can still provide a vibrotactile orother type of “haptics response” to such interactions by the user, i.e.pressure applied without deformation occurring.

FIG. 10 is a flow diagram illustrating the providing of haptic feedbackin a flexible device in accordance with some embodiments. FIG. 10illustrates a flow where an application requires a certain stiffness orflexibility of a flexible display which is static while the applicationis running. At 1010, display functionality is started on a flexibledisplay device. The flexible device can be bendable, foldable, rollable,or a combination thereof. At 1020, an application is started on theflexible display device that has the ability to request a certainelasticity. The application may be a process native to the operatingsystem, such as a home screen, settings screen, or notifications screen,or may be a user-installed application.

At 1030, the flow considers whether the application requires a highlyflexible interface. For example, the device could determine whether theintent is to bend the device (requiring flexibility) or interact withthe touchscreen (requiring rigidity) based on the location of the touchinput. When grasping the edges, the user may be about to bend thedevice, which should remain flexible. When touching closer to the middleof the screen, the user likely wants to interact with the screen and arigid surface may be better. The surface can also become rigid when theuser interacts with it using a stylus. If a high flexibility isrequired, at 1040, a haptic stiffness controller, such as actuator drivecircuit 16, can make the display more elastic or deformable. Greaterflexibility can be achieved by allowing the device to flex naturally orby augmenting the flexion of the device to assist flexion when a userapplies bending/folding/rolling force to make it seem “easier” to theuser to flex and manipulate the device. If at 1030, the applicationrequires low (or no) flexibility, then at 1050 the haptic stiffnesscontroller can make the display more rigid or completely rigid. Rigiditycan be achieved by controlling one or more actuators, such as actuator18, to resist deformation of the flexible display. When a user switchesapplications or when a function of the application changes, the flow canrepeat back to 1020 from either 1040 or 1050.

FIG. 11 is a flow diagram illustrating the providing of haptic feedbackin a flexible device in accordance with some embodiments. FIG. 11illustrates a flow where an application requires a stiffness orflexibility of a flexible display that is dynamically adjustable whilethe application at running based on a user's interaction with theapplication and device. At 1110, display functionality is started on aflexible display device. The flexible device can be bendable, foldable,rollable, or a combination thereof. At 1120, an application is startedon the flexible display device that has the ability to request a certainelasticity of the display dynamically throughout the course of theapplication execution. The application may be a process native to theoperating system, such as a home screen, settings screen, ornotifications screen, or may be a user-installed application. At 1130,an input or gesture is detected by the device through flexible display717, touch interface 11, or other input interface. The input is measuredand characterized by the application. At 1140, the elasticity of thedisplay is adjusted based on the user's input and the applicationsettings. Elasticity can be adjusted to be more rigid or more flexible.Rigidity can be controlled by a haptic stiffness controller, such asactuator drive circuit 16, to control one or more actuators, such asactuator 18, to resist deformation or assist in deformation of theflexible display to make the display more or less rigid, respectively.In some embodiments, the haptic stiffness controller can be controlledto be passive to allow the elasticity to be the same as the naturalelasticity of the device. Flow can continue to step 1130 to continuedetecting further input and reacting accordingly.

In one embodiment consistent with the flow illustrated in FIG. 10, therigidity of a flexible display, such as display 711 can be adjusted tobe stiffer when a user is inputting text. Applications implementing textentry require the screen to be rigid, otherwise the display deforms awayfrom user's finger every time they want to press a key. Moreover, theshape of the device and spatial arrangements of the keys should remainfixed during the keyboard entry for muscle memory to be effective duringwriting. As soon as the user launches the keyboard entry application,the haptic controller can stiffen the display, making it suitable forthe function (facilitating typing interactions).

In another embodiment consistent with the flow illustrated in FIG. 11,the flexible display can be equipped with embedded strain gauges andforce sensors, such as sensor 17, which can characterize the deformationand force inputs applied by the user, and a pressure-sensitive touchscreen, such as touch surface on 711, mounted on the flexible display.The device in its original mode can be rigid, allowing the user tointeract with different OS/interface functions. However, when the userapplies a bending force or touch pressure beyond than a certain value orthreshold, the device can switch to become deformable or less rigid.Therefore, the user can command the device to become soft when a softerinterface is required, but can take advantage of rigidity of the displayin other scenarios. Flexible devices can be damaged by excessivedeformation. Thus, in another example, haptic feedback can be used towarn users when they are about to exceed the maximum deformationsupported. The device can become rigid, and produce an abstract butunmistakable warning with vibration, or produce a squeaking sensationwith granular synthesis that intuitively communicates that it is bendingtoo much. This input monitoring and dynamic elasticity adjustmentfacilitates interactions with flexible displays across a variety offunctionalities, rendering the interface more effective for both theinteractions that require a soft display as well as those requiring astiffer one.

In another embodiment, the flexible display can be deformed andmaintained in a wearable configuration. In this embodiment, to be ableto wear the device on certain location of the body (e.g., wrist)ergonomically, the user can deform the device to make it conform to theshape of the body (e.g., bends it around his/her wrist). However, oncethe device is bent around the wrist, it can be made to become rigid orstiff to stay in place as a wearable portable device. Increasing thestiffness of the display once it is taken the target shape allows theuser to make the display into physical shapes that are appropriate forcertain functions, and make the display hold its shape for whatever timeis required.

In another embodiment, the flexible display can be deformed according toa gaming console application that can be wearable or take on a shapesuitable for interaction with the game. Here, the flexible display, suchas 711, can be separated into a flexible interface and a separatedisplay, where the flexible interface is an interface to a gamingconsole application.

Turning back to FIG. 7, another embodiment is a system that providesshape changing functions of a flexible display. Changing the shape ofthe display can depend on the type of interaction, as well as the typeand the content of the information that is communicated to the user. Inone embodiment a flexible display, such as the display of 711, can behaptically augmented to change shape in a self-actuated controlledmanner. The purposes for taking a particular shape can include theability to take the form that is more appropriate for a certainfunctionality or application and thus facilitates the user interactionswith the interface; to display certain information through its geometricshape; to enable functionalities that are only feasible with a certainphysical shape; or in cases where the device is serving as a physicalrepresentation of a virtual object, to deliver a more realisticsimulation of the target object by mimicking its shape.

Where shape change indicates information, one can inspect the deviceeither visually or haptically to view or experience the shape change toreceive the information (without the need to discern through detailedexamination visually or aurally). For example, a device that is flat mayhave no new messages or notifications while a device that is bent mayhave new messages where the degree of bending indicates the number ofmessages. This can be visually seen without inspecting in detail or byinteracting with the device, for example in a user's pocket.

FIG. 12 is a flow diagram illustrating the providing of haptic feedbackin a flexible device in accordance with some embodiments. At 1210,display functionality is started on a flexible display device. Theflexible device can be bendable, foldable, rollable, or a combinationthereof. At 1220, an application is started, resumed, or notified on theflexible display device which requires displaying information orrepresents a status change and that has the ability to request a certainshape of the device. The application may be a process native to theoperating system, such as a home screen, settings screen, ornotifications screen, or may be a user-installed application. At 1230,the flexible display device changes shape by itself to hapticallydisplay information or status of the device, or to take a formappropriate for the function.

For example, in one embodiment the flexible display can be a bendable orfoldable phone which bends or folds when there is an urgent incomingcall. The call would arrive and the phone could recognize the caller ordetermine urgency, and change its shape to signify the urgency of thecall or person calling. Similarly, the flexible display can change itsshape in a way that the physical form of the device better matches thecontent/info displayed. For example, the user can initiate a photobrowsing application. As soon as this application is initiated, theflexible device can take the form of a picture frame with a stand, tomore realistically simulate a photo viewing interaction (making the userexperience closer to real world interactions), and also facilitate thefunctionality for the use.

Referring again to FIG. 7, another embodiment includes bend-sensors andactuators, such as sensor 17 and actuator 18. In one embodiment, aflexible display, such as LG Corporation's “G Flex” phone can have abend sensor attached or integrated into its design, such as sensor 17,as well as sensor reading and processing circuitry such as processor 12and so forth. The “G Flex” is a curved phone that can slightly bend andincludes an integral vibrotactile haptic device. The bending sensorvalues can be sensed using an interface directly connected or connectedvia Bluetooth to the phone or display. Mounting an actuator directly ona flexible surface can prevent it from bending at the mounting location,or can cause the mounting to fail. For example, relatively largeactuators can become unglued or unattached from the flexible surfaceover time. This issue can be overcome by attaching a much smallerfixture (one with less surface area contact with the bending surface tooppose the bending direction) to the flexible surface, and thenattaching the actuator to it, thereby positioning the larger actuatoraway from the surface so that the actuator does not inhibit the bendingsurface. One solution can be to use a stud or post attached to theflexible device and glue or attach the actuator to the stud or posthead. In some embodiments the actuator can slide up the post or stud asthe bending surface begins to contact the actuator. Thus, a small, rigidmounting fixture can securely attach actuators without hindering thebending of the surface. In devices that bend in only one direction thesmall fixture can be thin in the direction perpendicular to the bendingdirection. In devices that bend in all directions, the small fixture canbe circular in shape at the contact point.

In another embodiment, actuators, sensors, and so forth from FIG. 7 canbe integrated into a flexible surface as an input device to a display.In other words, the flexible display of 711 can be split into a flexiblesurface and a display (not shown). A user can gesture with the inputdevice by deforming the surface, including bending or twisting thesurface or bending the corners. Graphics can be shown on a nearby phoneor display by transmitting the gestures to the display, wirelessly orwired. The graphics can be controlled and updated by deforming theflexible surface.

In embodiments using either the flexible surface or flexible display,such as the “G Flex”, haptic feedback can also be generated on theflexible surface or display, as in the following examples. A short burstof vibration can be emitted when a bending threshold is reached,creating the sensation of a “pop,” as if the surface was bi-stable andhad snapped into another position. This idea can be used to indicate anysudden change in state, such as breaking an object in two, like withsnapping a glow stick, a Hershey's “Kit Kat,” and so forth. In anotherexample, short bursts of vibration can be emitted at specific points inthe bending range to give the impression of detents. For example,detents could be emitted when the surface bends by 25%, 50%, 75% and100%.

In another example, granular synthesis can be used to create theimpression of creaking or to provide the illusion that the surface isbending more than it actually is by outputting a short burst ofvibration every time the bending of the surface changes by a particulargranular amount. Short bursts of vibration (e.g., 10 ms) can beoutputted every time the bend changes by a specific amount (e.g., 30grains over the range of bending). This can be done to create at leasttwo effects. First, the settings can be adjusted to simulate differenttextures. This can give the impression, for example, that the user isbending a wood, plastic, or glass surface of different thicknesses.Second, the creaking can also give the impression that the surface isbending more than it really is. This is important in transitionaldevices as flexible devices become more widely available and then moreflexible. A flexible but stiff device can feel as if it was much moreflexible than it really is.

One embodiment using these techniques includes a glow stick shown on thescreen. As the surface or device is bent, it first creaks using a hapticplayback device. As the glow stick creaks and the surface or device isbent, the display of the glow stick shows the glow stick starting tobend in accordance with the sensed bend pressure provided by the user.When the illustrated glow stick can bend no more without snapping, asnapping sensation is felt using a haptic playback device. Theapplication responds and the glow stick lights up.

Another embodiment using these techniques includes a camera app shown onthe screen. Bending the phone display or surface can control either thefocus or the zoom. Continuous haptic effects can be outputted to emulatethe mechanics of the camera.

Another embodiment using these techniques in a slightly bendable device,such as the “G Flex” or bendable surface, includes an e-book app shownon screen. Pages can be flipped one by one when bending the edges orcorners of the surface. Confirmation feedback can be outputted on eachflip. Alternatively, the surface can be bent and held in place, whichcan cause the pages to continuously flip at a fast rate. A haptic effectcan be played to allow the user to feel the flipping of the pages.Further, similar to earlier in the application, the application canclose or open the book when the device is bent one way or the other pasta threshold.

As disclosed, embodiments implement a kinesthetic and vibrotactilehaptic feedback system in flexible devices, including afoldable-bendable display device. Deformation input associated with auser's actions may be used, based on an application running on thedevice, to determine deformation characteristics of the device andthereby provide appropriate kinesthetic and vibrotactile feedback to theuser. Such feedback includes effects useful when using the device as ane-reader, such as a single page flip effect and a page browse effect.Such feedback also includes effects useful when comparing objects, oneon each flap of the display device. Such feedback also includes effectsuseful when opening and closing the folds of the device. Such feedbackalso includes augmentation of the device to increase or decrease devicerigidity or provide feedback based on an application on the device andthe deformation or deformation pressure placed on the device.

Several embodiments are specifically illustrated and/or describedherein. However, it will be appreciated that modifications andvariations of the disclosed embodiments are covered by the aboveteachings and within the purview of the appended claims withoutdeparting from the spirit and intended scope of the invention.

What is claimed is:
 1. A haptic effect enabled bendable-foldable displaycomprising: a first flexible flap and a second flexible flap coupled bya hinge; a haptic output device; a sensor electronically coupled to thefirst flap and the second flap to receive a folding characteristicsignal based on a value of an opening angle between the first flap andthe second flap, and to receive a bending characteristic signal of thefirst flap based on an amount of a bend radius of the first flap; and adrive circuit electronically coupled to an actuator configured toproduce a haptic effect based on the folding characteristic signal andthe bending characteristic signal, wherein the bending characteristicsignal corresponds to an action of flexing a bendable portion of thefirst flap and the folding characteristic signal corresponds to anaction of opening the hinge; and wherein the haptic output deviceprovides kinesthetic feedback of the hinge in response to the foldingcharacteristic signal and kinesthetic feedback of the first flap inresponse to the bending characteristic signal.
 2. The bendable-foldabledisplay of claim 1, wherein the drive circuit, based on the sensordetecting a sequence of signals comprising a touch pressure, an increasein the bend radius, and a decrease of the opening angle, generates aparticular haptic feedback.
 3. The bendable-foldable display of claim 1,wherein the kinesthetic feedback of the hinge modulates stiffness of theopening based on application settings for an application on the deviceand the folding characteristic signal; and wherein the kinestheticfeedback of the first flap modulates stiffness of the flexing based onapplication settings of the display and the bending characteristicsignal.
 4. The bendable-foldable display of claim 1: further comprising:memory to store the folding characteristic signal as a pre-fold statussignal prior to a change in the folding characteristic signal, whereinafter the change in the folding characteristic signal, the drive circuitcontrols the hinge to produce approximately the pre-fold status signalas the folding characteristic signal, and wherein an application on thedevice provides a visual change corresponding to the changing foldingcharacteristic signal.
 5. The bendable-foldable display of claim 1,further comprising: memory to store the folding characteristic signal asa post-fold status signal after a change in the folding characteristicsignal; memory to store the bending characteristic signal as a post-bendstatus signal after a change in the bending characteristic signal,wherein upon a change in the folding characteristic signal and change inthe bending characteristic signal indicate a minimal value of bendingand a small opening in the hinge, the drive circuit controls the hingeto produce approximately the post-fold status signal as the foldingcharacteristic, and wherein an application on the device provides avisual change corresponding to the changing folding characteristicsignal.
 6. The bendable-foldable display of claim 1, wherein the drivecircuit controls the device in response to an application on the devicethat displays an object on the first flap of the display, wherein theapplication contains physical property information about each objectdisplayed, wherein the drive circuit provides kinesthetic feedback inresponse to the bending characteristic signal of the first flap tostiffen or relax bendability of the first flap based on the physicalproperty information of the object displayed on the first flap.
 7. Thebendable-foldable display of claim 6, wherein the drive circuit further:controls vibrotactile feedback in response to the application on thedevice, wherein the vibrotactile feedback provides a haptic responsebased on input from a touch sensor.
 8. The bendable-foldable display ofclaim 1, wherein the folding characteristic signal changes indicating achange in position of the hinge indicating that the foldable portion ofthe device is closing, and wherein the drive circuit produces, prior tothe closing of the foldable portion, kinesthetic feedback to control thefoldable portion closed.
 9. The bendable-foldable display of claim 1,wherein the folding characteristic signal changes indicating a change inposition of the hinge indicating that the foldable portion of the deviceis opening, and wherein the drive circuit produces, prior to the openingof the foldable portion, kinesthetic feedback to control the foldableportion open.
 10. The bendable-foldable display of claim 1, wherein thefolding characteristic signal changes indicating a change in position ofthe hinge indicating that the foldable portion of the device is closing,and wherein the folding characteristic signal changes indicating afurther change in the position of the hinge indicating that the hinge isclosed, and wherein the drive circuit produces haptic feedback on thehaptic output device.
 11. The bendable-foldable display device of claim1, wherein the folding characteristic signal indicates that the hinge isfully closed, wherein the folding characteristic signal changes toindicate that the hinge is opening, and wherein the drive circuitproduces haptic feedback on the haptic output device.
 12. A hapticsystem comprising one or more haptic output devices integrated into abendable-foldable display comprising: sensor information receivers forreceiving input from one or more sensors indicating deformation of thebendable-foldable display device having a first flexible flap and asecond flexible flap coupled by a hinge, wherein the one or more sensorsare configured to detect a folding characteristic based on a value of anopening angle between the first flexible flap and the second flexibleflap and configured to detect a bending characteristic in at least oneflap based on an amount of bend radius of the at least one flap; aprocessor for determining deformation characteristics of thebendable-foldable display device, wherein the determination is based onthe folding characteristic and the bending characteristic; and a hapticfeedback generator that generates a haptic feedback based on thedeformation characteristics, wherein the deformation characteristicscorrespond to an action of flexing a bendable portion of the display andopening a foldable portion of the display; and wherein the hapticfeedback includes kinesthetic feedback of the foldable portion of thedisplay and kinesthetic feedback of the bendable portion of the display.13. The system device of claim 12, wherein the haptic feedbackgenerator, based on the sensor detecting a sequence of signalscomprising a touch pressure, an increase in the bend radius, and adecrease of the opening angle, generates a particular haptic feedback.14. The system of claim 12, wherein the kinesthetic feedback of thefoldable portion of display modulates stiffness of the opening based onapplication settings for an application on the device and thedeformation characteristics; and wherein the kinesthetic feedback of thebendable portion of the display modulates stiffness of the flexing basedon the application settings and deformation characteristics.
 15. Thesystem claim 12, wherein: the processor stores pre-deformation positioninput from the one or more sensors prior to the deformation; and thehaptic feedback generator: controls one or more haptic actuators toreturn the foldable-bendable display near to its pre-deformationposition based on the pre-deformation input; and causes an applicationon the device to provide a visual change corresponding to thedeformation characteristics.
 16. The system of claim 12, wherein: theprocessor: stores post-deformation input from the one or more sensorscorresponding to a position value for the opening and a flexion valuefor the flexing, wherein the flexion value indicates a flexing of thebendable display; receives input indicating a change in opening andflexion deformation characteristics, wherein the changed deformationcharacteristics indicate a minimal value of flexion and a smalleropening at the foldable portion; and the haptic feedback generator:controls one or more haptic actuators to return the foldable-bendabledisplay to an opening position value near to its post-deformationopening position value; and causes an application on the device toprovide a visual change corresponding to the deformationcharacteristics.
 17. The system of claim 12, wherein the haptic feedbackgenerator: controls kinesthetic feedback in response to an applicationon the device that displays an object on each flap of thefoldable-bendable display, wherein the application contains physicalproperty information about each object displayed, wherein thekinesthetic feedback provides a modulated response to deformationcharacteristics of the bendable flap to stiffen or relax bendability ofthe flaps based on the physical property information of each object. 18.The system of claim 17, wherein the haptic feedback generator: controlsvibrotactile feedback in response to the application on the device,wherein the vibrotactile feedback provides a haptic response based oninput from a touch sensor.
 19. The system of claim 12, wherein: theprocessor receives input indicating a change in position of each flap,wherein the change indicates that the foldable portion of the device isclosing; and the haptic feedback generator produces, prior to theclosing of the foldable portion, kinesthetic feedback to control thefoldable portion closed.
 20. The system of claim 12, wherein: theprocessor receives input indicating a change in position of each flap,wherein the change indicates that the foldable portion of the device isopening; and the haptic feedback generator produces, prior to theopening of the foldable portion, kinesthetic feedback to control thefoldable portion open.
 21. The system of claim 12, wherein: theprocessor receives input indicating a change in position of each flap,wherein the change indicates that the foldable portion of the device isclosing; the processor receives input that the flaps are fully closed;and the haptic feedback generator produces haptic feedback on the one ormore haptic output devices.
 22. The system of claim 13, wherein: theprocessor receives input that the flaps are fully closed; the processorreceives input indicating a change in position of each flap, wherein thechange indicates that the foldable portion of the device is opening; andthe haptic feedback generator produces haptic feedback on the one ormore haptic output devices.
 23. A flexible-foldable display devicecomprising: a first flexible flap and a second flexible flap coupled bya hinge; one or more sensors for detecting a value of an opening anglebetween the first flap and the second flap; a haptic system comprisingone or more haptic output devices, wherein the haptic system: receivesinput from the one or more sensors indicating the value of the openingangle between the first flap and the second flap and generates hapticfeedback based on the value of the opening angle, wherein the hapticfeedback comprises a first and a second vibrotactile effect, wherein thefirst vibrotactile effect is based on a first deformation characteristiccomprising a bendable portion of the first flap and the secondvibrotactile effect is based on a second deformation characteristiccomprising an action of opening the hinge, and wherein the first andsecond deformation characteristics are non-overlapping ranges ofdeformation.
 24. The flexible device of claim 23, wherein: the hapticfeedback is a kinesthetic feedback to control rigidity of the flexibledevice to make the device more or less stiff.
 25. The flexible device ofclaim 24, wherein: the rate of change of an image on the first flap orthe second flap is dependent on the opening angle.
 26. A haptic systemcomprising one or more haptic output devices integrated into a flexibledisplay comprising: a first flexible flap and a second flexible flapcoupled by a hinge; sensor information receivers for receiving inputfrom one or more sensors indicating a bending deformation of theflexible display device and a folding deformation between the firstflexible flap and the second flexible flap, wherein the one or moresensors are configured to determine an amount of bending radius in thedisplay and a value of an opening angle between the first flap and thesecond flap; and a haptic feedback generator that generates hapticfeedback based on the amount of bending radius and the value of theopening angle, wherein the haptic feedback comprises a first and asecond vibrotactile effect, wherein the first vibrotactile effect isbased on a first deformation characteristic comprising a bendableportion of the first flap and the second vibrotactile effect is based ona second deformation characteristic comprising an action of opening thehinge, and wherein the first and second deformation characteristics arenon-overlapping ranges of deformation.
 27. The system of claim 26,wherein: the haptic feedback includes a kinesthetic effect.
 28. Thesystem of claim 26, wherein: the haptic feedback is a kinestheticfeedback to control rigidity of the flexible device to make the devicemore or less stiff.
 29. The system of claim 28, wherein: the rigiditywas more stiff prior to characteristic detection of the bending radiusand less stiff after the detection of the bending radius.